Skip to content

fix!: resume a pipeline snapshot taken on a later loop visit - #12162

Merged
sjrl merged 15 commits into
mainfrom
minimal-fix-snapshot-resume
Jul 29, 2026
Merged

fix!: resume a pipeline snapshot taken on a later loop visit#12162
sjrl merged 15 commits into
mainfrom
minimal-fix-snapshot-resume

Conversation

@sjrl

@sjrl sjrl commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

Fixes

  • Fixed resuming a Pipeline from a pipeline_snapshot that was taken on a component's second or later visit,
    which failed with PipelineComponentsBlockedError: Cannot run pipeline - all components are blocked. A snapshot stored only the values of the pipeline's inputs and dropped the information about which component had sent each one, so on resume every restored input looked like it came from outside the pipeline, and such an input can only trigger a component on its first visit. Snapshots now record the sender of each input. Snapshots created by earlier versions of Haystack behave as before, so re-create them to resume anywhere in a looping pipeline.

  • Fixed a resumed Pipeline passing malformed inputs to the component the snapshot was taken on, whenever that component ran more than once after the resume, for example inside a loop. Every visit after the first reused the handling meant only for the paused visit and skipped the regular input consumption, so a variadic component could receive a bare value where it expected a list, raising errors such as TypeError: object of type 'int' has no len() from a BranchJoiner. This affected snapshots taken at any visit count, including the first.

  • The sentinel marking a socket whose sender produced no output for it is now its own type, _NoOutputProduced, rather than an alias of the _empty marker for an unset InputSocket.default_value. The two record unrelated things, and we needed to add serde support to _NoOutputProduced.

Breaking Change

  • PipelineSnapshot.pipeline_state.inputs changed shape. It used to store one flattened value per socket, {component: {socket: value}}. It now stores the pipeline's internal inputs, keeping the component that sent
    each one and keying the inputs a socket received by their position: {component: {socket: {"0": {"sender": ..., "value": ...}}}}. Positions are used instead of a list so that each input carries its own type schema, which a list cannot do.
  • PipelineState gained an inputs_format field recording which of the two shapes a snapshot uses. The same applies to BreakpointException.inputs, which returns that field.

You are affected if you read pipeline_state.inputs (or BreakpointException.inputs) directly, for example
to display or post-process a snapshot. Resuming a pipeline with Pipeline.run(pipeline_snapshot=...) is not
affected, and neither is code that only passes snapshots around or persists them.

To adapt, read the input through its position and take its value:

inputs = snapshot.pipeline_state.inputs["serialized_data"]

# before
value = inputs["my_component"]["my_socket"]

# now
value = inputs["my_component"]["my_socket"]["0"]["value"]

A socket that received inputs from several senders now has one entry per position, "0", "1" and so on,
each recording the sender that produced it.

Snapshots written by earlier versions of Haystack have inputs_format set to None and keep the flattened
shape, so branch on that field if you need to handle both.

How did you test it?

New tests and updated existing ones.

Notes for the reviewer

This has a breaking change associated with it in addition to being a fix. The inputs format has changed so anyone who relied on inspecting on the old format will get errors. More details in the reno.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

…pipelines. This fixes a bug when using a pipeline snapshots with pipelines that contain loops.
@vercel

vercel Bot commented Jul 27, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
haystack-docs Ready Ready Preview, Comment Jul 29, 2026 5:58am

Request Review

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/core/component
  types.py
  haystack/core/pipeline
  base.py
  breakpoint.py
  component_checks.py
  pipeline.py
  haystack/dataclasses
  breakpoints.py
Project Total  

This report was generated by python-coverage-comment-action

Comment on lines +222 to +225
The inputs a socket received are stored keyed by position rather than as a list, because a list gets a single
schema derived from its first item. A socket with several senders can hold values of different types, for
example a value from one sender and a `_NoOutputProduced` marker from another, and those need one schema
each to survive the round trip.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This explains why this function is needed as a workaround since _serialize_with_field_fallback does not support serializing mixed-type lists.

Comment on lines +241 to +243
def _deserialize_internal_inputs(serialized_inputs: dict[str, Any]) -> dict[str, Any]:
"""
Restore the pipeline's internal inputs state from a snapshot written by `_serialize_internal_inputs`.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to comment https://github.com/deepset-ai/haystack/pull/12162/changes#r3656582668 that this function is only needed to work around _deserialize_value_with_schema not supporting deserializing mixed type lists.

Comment on lines +10 to +12
class _NoOutputProduced:
"""
Marker a socket input holds when its sender ran without producing a value for that socket.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was upgraded to a proper standalone class b/c it must support serde since it can be included in the serialized inputs in PipelineState.inputs

pipeline.run(data={"comp1": {"input_value": "test"}}, break_point=Breakpoint(component_name="comp2"))
snapshot = exc_info.value.pipeline_snapshot
@pytest.mark.parametrize("visit_count", [0, 1, 2, 3])
def test_break_point_in_loop_resumes_on_any_visit(self, visit_count):

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Start of new tests

@sjrl
sjrl marked this pull request as ready for review July 27, 2026 12:20
@sjrl
sjrl requested a review from a team as a code owner July 27, 2026 12:20
@sjrl
sjrl requested review from davidsbatista and removed request for a team July 27, 2026 12:20
@davidsbatista davidsbatista changed the title fix: resume a pipeline snapshot taken on a later loop visit !fix: resume a pipeline snapshot taken on a later loop visit Jul 28, 2026
@davidsbatista

Copy link
Copy Markdown
Contributor

added ! to PR's title since it's a breaking change



class TestCreatePipelineSnapshot:
def test_create_pipeline_snapshot_all_fields(self):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe worth a "live" test for this one ?

@davidsbatista

Copy link
Copy Markdown
Contributor

overall looks good, left a comment regarding the tests

@davidsbatista davidsbatista left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, left a suggestion for an extra test, not blocking

@sjrl sjrl changed the title !fix: resume a pipeline snapshot taken on a later loop visit fix!: resume a pipeline snapshot taken on a later loop visit Jul 29, 2026
@sjrl
sjrl enabled auto-merge (squash) July 29, 2026 05:56
@sjrl
sjrl merged commit 66fb9d0 into main Jul 29, 2026
25 checks passed
@sjrl
sjrl deleted the minimal-fix-snapshot-resume branch July 29, 2026 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resuming a pipeline snapshot after the first loop visit fails with PipelineComponentsBlockedError

2 participants